Skip to content

ci(bench): add local e2e benchmark harness#3822

Open
joshieDo wants to merge 17 commits intomainfrom
codex/e2e-bench-harness
Open

ci(bench): add local e2e benchmark harness#3822
joshieDo wants to merge 17 commits intomainfrom
codex/e2e-bench-harness

Conversation

@joshieDo
Copy link
Copy Markdown
Contributor

@joshieDo joshieDo commented May 5, 2026 •
edited
Loading

Adds a single-runner e2e benchmark workflow for comparing baseline and feature refs over a two-validator localnet. It initializes/restores schelk snapshots for both validators, defaults e2e to txgen on 100g bloat, and keeps samply/Tracy, Grafana, ClickHouse, PR, and Slack result posting wired through.

Changelog for e2e users:

  • bloat is now 1g, 10g, or 100g (default 100g), not raw MiB.
  • backend=txgen is the default; use backend=tempo-bench for the legacy sender.
  • Hardfork comparisons are currently rejected by the local e2e harness.
  • Samply profiles are posted per phase and validator (a/b).
  • Grafana observability links are wired but not working reliably yet.

Validated with: https://github.com/tempoxyz/tempo/actions/runs/25506968095/attempts/1#summary-74855167179

@joshieDo joshieDo marked this pull request as draft May 5, 2026 15:29
@joshieDo joshieDo changed the title ci(bench): add two-validator e2e harness ci(bench): add local e2e benchmark harness May 7, 2026
@joshieDo joshieDo force-pushed the codex/e2e-bench-harness branch from b122cc8 to 7a2c387 Compare May 7, 2026 10:04
@decofe
Copy link
Copy Markdown
Member

decofe commented May 7, 2026

@decofe bench duration=10 tps=10 bloat=1g backend=txgen samply

@decofe
Copy link
Copy Markdown
Member

decofe commented May 7, 2026

❌ Invalid bench command

Invalid value(s): bloat=1g (must be a positive integer)

Usage: @decofe bench [mode=MODE] [chain=mainnet|testnet] [blocks=N] [warmup=N] [preset=NAME] [duration=N] [bloat=N] [tps=N] [baseline=REF] [feature=REF] [backend=NAME] [txgen-ref=REF] [samply] [force-bloat] [no-slack] [existing-recipients=BOOL] [tracy=MODE] [tracy-seconds=N] [tracy-offset=N] [baseline-args="ARGS"] [feature-args="ARGS"] [baseline-hardfork=FORK] [feature-hardfork=FORK] [bench-args="ARGS"] [bench-env="VARS"] [baseline-env="VARS"] [feature-env="VARS"]

1 similar comment
@decofe
Copy link
Copy Markdown
Member

decofe commented May 7, 2026

❌ Invalid bench command

Invalid value(s): bloat=1g (must be a positive integer)

Usage: @decofe bench [mode=MODE] [chain=mainnet|testnet] [blocks=N] [warmup=N] [preset=NAME] [duration=N] [bloat=N] [tps=N] [baseline=REF] [feature=REF] [backend=NAME] [txgen-ref=REF] [samply] [force-bloat] [no-slack] [existing-recipients=BOOL] [tracy=MODE] [tracy-seconds=N] [tracy-offset=N] [baseline-args="ARGS"] [feature-args="ARGS"] [baseline-hardfork=FORK] [feature-hardfork=FORK] [bench-args="ARGS"] [bench-env="VARS"] [baseline-env="VARS"] [feature-env="VARS"]

@joshieDo joshieDo marked this pull request as ready for review May 7, 2026 16:46
@joshieDo joshieDo requested a review from shekhirin May 7, 2026 16:47
shekhirin
shekhirin reviewed May 7, 2026
View reviewed changes
Comment thread .github/workflows/bench-e2e.yml
default: "300"
bloat:
description: State bloat size in MiB.
description: State bloat snapshot size (1g, 10g, 100g).
Copy link
Copy Markdown
Member

@shekhirin shekhirin May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we make it gb otherwise i will think it's grams

Copy link
Copy Markdown
Member

@shekhirin shekhirin May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and the value itself i think we can just pass the int? so we don't need to strip gb from it

shekhirin
shekhirin reviewed May 7, 2026
View reviewed changes
Comment thread .github/workflows/bench-e2e.yml
type: choice
required: true
default: tempo-bench
default: txgen
Copy link
Copy Markdown
Member

@shekhirin shekhirin May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine if it works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@shekhirin shekhirin shekhirin approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@joshieDo @decofe @shekhirin